Gaze target application launcher

ABSTRACT

A gaze target is recognized via an eye tracking camera. An application launcher is displayed, via a display, at the gaze target based on a user trigger. The application launcher presents a plurality of applications selectable for launching. A user selection of one of the plurality of applications is recognized. The application launcher is replaced with the selected one of the plurality of applications at the gaze target via the display.

BACKGROUND

Computing devices may execute applications for carrying out operations.In order to access the applications of a computing device, a userinterface element with indications of available applications may bepresented for selection.

SUMMARY

Embodiments are disclosed for controlling a location of an applicationlauncher for a computing device. An example method includes recognizinga gaze target, and, responsive to a user trigger, displaying anapplication launcher at the gaze target. The method further includesrecognizing a user selection of one of the plurality of applications,and replacing the application launcher with the selected one of theplurality of applications at the gaze target.

This Summary is provided to introduce a selection of concepts in asimplified form that are further described below in the DetailedDescription. This Summary is not intended to identify key features oressential features of the claimed subject matter, nor is it intended tobe used to limit the scope of the claimed subject matter. Furthermore,the claimed subject matter is not limited to implementations that solveany or all disadvantages noted in any part of this disclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows an example environment for interacting with a computingdevice and associated display.

FIG. 2 is a flow chart of an example method for controlling a positionof an application launcher.

FIG. 3 shows an example field of view of a user of a head-mounteddisplay device in the example environment of FIG. 1.

FIGS. 4A-4B show an example timeline of a field of view of a userthrough a head-mounted display.

FIG. 5 shows an example environment for presenting an applicationlauncher on a display device.

FIG. 6 is a block diagram of an example computing system.

FIG. 7 schematically shows an example of a head-mounted display device.

DETAILED DESCRIPTION

Computing devices of the present disclosure are configured to facilitateapplication launching. In one example, an augmented reality computingdevice may recognize the gaze target at which a user is looking.Responsive to a user trigger, the augmented reality computing device isconfigured to display an augmented reality image of an applicationlauncher (or an application launcher affordance or preview) so that theapplication launcher appears to be located at the gaze target.Applications subsequently launched from the application launcher mayreplace the application launcher. As such, by selecting a location forthe application launcher, the user is effectively selecting a locationfor a subsequently launched application. Both selections are based onthe user's gaze. In another example, a user's gaze onto a conventionaldisplay is assessed, an application launcher or preview is displayed atthe display location of the gaze, and a launched application replacesthe application launcher responsive to user selection.

In augmented reality environments, the application launcher allows auser to launch applications “out of thin air” or as an overlay on anysuitable surface in the real-world environment without leaving theaugmented reality experience. In such environments, the applicationlauncher may be world-locked, such that the launcher appears to stay inthe same position relative to real-world objects, regardless of themovements of the head-mounted display device. In other implementations,the application launcher may be body-locked, such that it appears tomove with the user. In some implementations, an application launcher canbe switched between world-locked and body-locked modes.

FIG. 1 shows an example environment 100 in which a user 102 may interactwith one or more of computing device 104 and head-mounted display device106 to launch applications. For example, applications may be launchedusing the head-mounted display device 106. Computing device 104 mayinclude any suitable computing device, including but not limited to avideo game console, a home entertainment device (e.g., an audio/videoreceiver), a personal computing device, a laptop, a smart phone, atablet computing device, and/or any other suitable computing system.Head-mounted display device 106 may include one or more output devices,such as see-through display(s) for presenting display data as anaugmentation to a real-world environment viewable through thesee-through display(s) and speaker(s) for outputting audio data.Head-mounted display (HMD) device 106 may further include one or moresensor subsystems integrated into the HMD device or communicativelycoupled to the HMD device. For example, the one or more sensorsubsystems may include but are not limited to an eye-tracking camera(e.g., a rearward-facing camera configured to capture images of an eyeof a wearer of the head-mounted display device to determine a gazetarget of the wearer), a microphone, and a forward-facing camera (e.g.,for capturing images of an environment).

Computing device 104 and/or head-mounted display device 106 may becommunicatively connected to display device 108 and/or the see-throughdisplay(s) of head-mounted display device 106 for presenting anapplication launcher and/or one or more launched applications. In otherimplementations, application launching functionality may be handled byan on-board computing system of the display device 108 and/or thehead-mounted display device 106. In still other implementations, atleast some of the application launching functionality may be handled bya remote computer or service. An example application launcher 109 isshown on a wall of the environment, the dashed lines indicating that theapplication launcher is viewable as an augmentation to the environmentthrough the head-mounted display device 106. Application launcher 109may include one or more icons or other user interface elementsindicating one or more applications that may be launched via selectionof the respective user interface element.

Sensor 110 may include one or more of a visible light camera, aninfrared camera, and/or a visible/infrared light projector for capturingRGB image data/frames and/or depth image data/frames (e.g., ofenvironment 100). In some examples, one or more cameras of sensor 110may be eye-tracking cameras configured to capture images of a user's eyefor determining a gaze target of the user. Sensor 110 may additionallyor alternatively include one or more directional microphones forreceiving sound (e.g., from environment 100).

FIG. 2 is a flow chart of a method 200 for controlling a position of anapplication launcher. Method 200 may be performed by any suitablecomputing device, including but not limited to computing device 104,head-mounted display device 106, or a remote computing device orservice. At 202, method 200 includes recognizing a gaze target. Anysuitable gaze tracking technique may be performed to recognize the gazetarget. For example, the eye-tracking camera of head-mounted displaydevice 106 and/or an eye-tracking camera of sensor 110 of FIG. 1 maycapture images of a user's eye in order to determine a gaze direction.The gaze direction may include a vector extending from the user's eyetoward a display or surface (e.g., a see-through display of head-mounteddisplay device 106, display device 108 of FIG. 1, and/or a surface onwhich a projector may project images). The location (e.g., a pixelcoordinate) at which the vector of the gaze direction intersects thedisplay may be determined to be the gaze target (e.g., the location onthe display at which the gaze is directed). In other examples, a gazetarget may be recognized by tracking a head movement. In such examples,the term “gaze” as used herein may refer to a head-targeting gaze.

At 204, the method includes recognizing a user trigger indicating arequest to display an application launcher. The user trigger may includeany suitable user input recognizable by the computing device and/orhead-mounted display device. For example, the user trigger may include agesture (e.g., a hand, face, and/or body gesture) recognized via one ormore of the eye-tracking camera (e.g., as used for recognizing the gazetarget) and another imaging device (e.g., a visible light, infrared, ordepth camera for imaging the environment, such as sensor 110 of FIG. 1).The user trigger may additionally or alternatively include a voicecommand recognized via a microphone and one or more voice recognitionmodules (e.g., local to the computing device and/or within a remoteservice communicatively connected to the computing device). Inadditional or alternative examples, the user trigger may include asustained gaze (e.g., a recognition that a user gaze is directed to thegaze target and/or the gaze target is maintained for a duration that isgreater than a trigger threshold). A user trigger including a gazetarget recognition may include identifying that the gaze is directed toan “empty” region of a display, user interface, and/or environment. Forexample, an “empty” region may include a region that does not include auser interface element and/or a region that does not include aselectable real-world or displayed object.

The user trigger may additionally or alternatively include a gazedirected at a user interface element displayed on the display device, aphysical button actuation on a control device or other input device(e.g., a control device external to the head-mounted display and/or thecomputing device used to launch applications), and/or any other suitableuser input. It is to be understood that any combination of user inputsmay be recognized as a user trigger.

As indicated at 206, the method may include displaying a positionalpreview of an application launcher (e.g., application launcher 109 ofFIG. 1) at the gaze target responsive to a user trigger. Displaying apositional preview of the application launcher may provide anopportunity for the user to fine tune the placement of the applicationlauncher. The positional preview may also allow the user to confirm thatthe placement of an application launcher in the location indicated bythe preview is the intended result of the gaze target and user trigger.For example, the positional preview may be an outline of an applicationlauncher (e.g., an outline of the outer edges of the user interfaceelement or application window corresponding to the applicationlauncher). In additional or alternative examples, the positional previewmay be a semi-transparent version of the application launcher, ananimation of the application launcher, and/or an otherwise modifiedversion of the application launcher (e.g., having a different size,rotation, shape, and/or other feature) that provides an indication of aposition, size, and/or shape with which the application launcher will bedisplayed. In some examples, a center of the positional preview may beat a same pixel coordinate in the display as the application launcher(once the positional preview is accepted). In this way, the preview mayprovide an approximation of the appearance, size, position, and/or otherfeature of the to-be-displayed application launcher and/or application(e.g., to allow a user to determine an amount of space the applicationthat will be launched will occupy).

As indicated at 208, method 200 may include determining if aconfirmation input or an expiration of the preview duration has beenrecognized. The confirmation input may include any suitable user input,including but not limited to one or more of the examples described abovewith respect to the user trigger. For example, the confirmation inputmay include a gesture recognizable by a camera integrated in and/orremote to a head-mounted display. The positional preview mayadditionally or alternatively have a predefined (e.g., defined by amanufacturer and/or within user-controlled settings) preview durationdefining an amount of time to display the preview absent user inputconfirming/cancelling/adjusting the preview.

If no confirmation input is recognized and the preview duration has notexpired at 208 (e.g., “NO” at 208), the method may proceed to 210 todetermine whether a cancellation or adjustment input is recognized. Acancellation input may include a suitable user input (e.g., anycombination of one or more of the above-described user input examples)indicating an intent to cancel presentation of the application launcher(and the associated positional preview). For example, a cancellationinput may allow a user to indicate a false positive user triggerrecognition. An adjustment input may include a different user input fromthe cancellation input, which identifies a user intent to adjust aposition of the positional preview (and a subsequently-displayedapplication launcher). An adjustment input may include a user inputindicating a direction in which to move the positional preview, anamount to move the positional preview, and/or a location on the displayat which an edge, center, and/or other portion of the positional previewis to be placed. For example, a user may perform a gesture indicating anadjustment is to be made, and gaze at a new location at which the centerof the preview is to be placed.

If a cancellation and/or adjustment input is not received at 210 (e.g.,“NO” at 210), the method may proceed to 212 to continue displaying thepreview at the current location and return to 208 to continue checkingfor a confirmation input and/or expiration of the preview duration.Conversely, if a cancellation and/or adjustment input is received at 210(e.g., “YES” at 210), the method may proceed to 214 to stop displayingthe preview at the current location. For example, if a cancellationinput is recognized at 210, the preview may be completely removed fromthe display, and the method may end and/or return to 202 to recognize anew gaze target. If an adjustment input is recognized at 210, the methodmay include moving the preview to a new location, as indicated at 216,and return to 208 to determine if the new location is acceptable basedon a confirmation input and/or expiration of a preview duration.

Returning to 208, if a confirmation input or an expiration of thepreview duration is recognized (e.g., “YES” at 208), the method proceedsto 218 to display an application launcher. For example, the positionalpreview may be replaced with the application launcher as indicated at220, such that at least an edge, center, or other portion of theapplication launcher is placed at a respective location of an edge,center, or other portion of the positional preview (e.g., displayed at206 and/or in the new location at 216). It is to be understood that inexamples where a preview is not displayed (e.g., where 206-216 are notperformed), the application launcher may be displayed at 218 directlyresponsive to the user trigger recognized at 204. An example applicationlauncher 109 is shown in FIG. 1 and further examples are described belowwith respect to FIGS. 4A, 4B, and 5. It is also to be understood that insome examples, the preview duration may be associated with acancellation of the positional preview and application launcherdisplaying, rather than a confirmation. In such examples, an expirationof the preview duration at 208 may cause the method to proceed to 214,rather than 218.

The application launcher displayed at 218 may present a plurality ofapplications selectable for launching. For example, the applicationlauncher may present all applications executable by the computing deviceand/or a subset of the applications executable by the computing device.In the latter example, the subset of applications may be automaticallyselected for presentation based on user preferences, a current contextof the computing device (e.g., an environment of the computing device, astatus of the computing device, a time of day, etc.), recent activity ofthe computing device (e.g., open or recently-used applications), thecurrent gaze target, user instruction, and/or any other suitableparameter. The application launcher may include more applications thanare viewable within the boundaries of the application launcher, andscrolling and/or page turning may be used to change the applicationscurrently presented for launching.

At 222, method 200 includes recognizing a user selection of anapplication in the application launcher. User selection of theapplication may be performed in any suitable manner and include any ofthe above-described user input mechanisms. For example, a user mayidentify a particular application by providing a voice commandidentifying that application and/or directing a gaze toward a displayeduser interface element within the application launcher identifying thatapplication. As another example, the user may perform a pointing gesturerecognized by one or more cameras.

As indicated at 224, the method includes replacing the applicationlauncher with the selected application. For example, replacing theapplication launcher with the selected application may includedisplaying a graphical user interface element and/or window of theselected application at the gaze target and/or at the location of theapplication launcher. The replacement of the application launcher withthe application may be similar to the optional replacement of thepositional preview with the application launcher, such that adisplayable portion of the application (e.g., an edge, center,boundaries, etc.) is aligned with a respective portion of theapplication launcher and displayed in place of the application launcher.As used herein, the term “launch” as applied to launching an applicationmay include executing an application and/or bringing an application intofocus/display (e.g., giving an already-executed/currently runningapplication system-wide focus or otherwise bringing the application intoview on the display in the location defined by the applicationlauncher).

FIG. 3 shows an example field of view 300 of user 102 of head-mounteddisplay device 106 within environment 100 of FIG. 1. For example, fieldof view 300 may represent a portion of environment 100 viewable throughsee-through display(s) of head-mounted display device 106. Within thefield of view, the user may be gazing in a direction indicated by gazeline 302, which may correspond to a gaze target 304. For example, if theuser intends to launch an application, the empty wall located at gazetarget 304 may be selected.

FIGS. 4A-4B show an example timeline 400 of fields of view through asee-through display of a head-mounted display (e.g., user 102 ofhead-mounted display device 106 in FIGS. 1 and 3). For example, withinthe field of view at time T1, real-world objects in environment 100,such as computing device 104, display device 108, and sensor device 110of FIG. 1, may be viewable alongside displayed augmentation elements,such as notification icons 402 (e.g., showing a weather and a newmessage indication). As represented by gaze line 404, the user's gazemay be directed toward gaze target 406, corresponding to a wall in thereal-world environment.

Before time T2, a user trigger is recognized, indicating an intent todisplay an application launcher at the gaze target 406. For example, theuser trigger may be performed and/or recognized at some time betweentimes T1 and T2. Accordingly, a positional preview 408 of theapplication launcher may be displayed as an augmentation to thereal-world environment, such that the preview is centered at the gazetarget.

Before time T3, a confirmation input is recognized, indicating aconfirmation that the positional preview is representative of anintended location of the application launcher. For example, theconfirmation input may have been performed and/or recognized at somepoint between times T2 and T3. Accordingly, the positional preview 408is replaced with an application launcher 410. The application launcher410 includes a plurality of icons 412, each icon identifying a differentselectable application for launching. The application launcher occupiesthe same space previously occupied by the positional preview.

At time T4, a user gaze indicated by gaze line 414 is directed to aselected one of the icons 412 (e.g., icon 412 a). A user selection ofthe application is performed at some time between times T4 and T5.Accordingly, at time T5, the application launcher is replaced with theselected application (e.g., a user interface window 416 associated withthe selected application). In the illustrated example, the selectedapplication is presented within the same boundaries as the applicationlauncher, however other configurations may be provided. For example, thecenter of the user interface window 416 may be positioned at the samelocation as the center of the application launcher 410, but the launchedapplication may have a different size, orientation, and/or shape as theapplication launcher.

FIG. 5 shows an example scenario in which user 102 in environment 100 ofFIG. 1 interacts with computing device 104 and display device 108,rather than head-mounted display device 106 and associated see-throughdisplay(s). As illustrated, a gaze of user 102, represented by gaze line502, in combination with a user trigger may cause computing system 104to display application launcher 504 on display device 108. For example,sensor device 110 may image an eye of user 102 and transmit the imagesto computing device 104. Computing device 104 may analyze the imagesand/or transmit the images to a remote service (e.g., a cloud computingdevice) via a network for remote processing/analysis. Upon analyzing theimages, the computing device may determine a gaze direction and map thegaze direction to a location on display device 108 to determine a gazetarget. Responsive to a user trigger and/or a confirmation of apositional preview, the computing device 104 may instruct the displaydevice to display the application launcher at the location of the gazetarget (e.g., center the application launcher at the gaze target).

The above-described methods and systems may provide a seamlessapplication launching experience for a user of a computing device and/orhead-mounted display device. By allowing application launchers (andsubsequently selected applications) to be positioned in a location of auser's choosing, the user may ensure that new applications are able tobe launched without cumbersome rearrangements of existing displayedobjects or interference from real-world objects in an augmented realityenvironment.

In some embodiments, the methods and processes described herein may betied to a computing system of one or more computing devices. Inparticular, such methods and processes may be implemented as acomputer-application program or service, an application-programminginterface (API), a library, and/or other computer-program product.

FIG. 6 schematically shows a non-limiting embodiment of a computingsystem 600 that can enact one or more of the methods and processesdescribed above. Computing system 600 is shown in simplified form.Computing system 600 may take the form of one or more head-mounteddisplay devices, wearable devices, personal computers, server computers,tablet computers, home-entertainment computers, network computingdevices, gaming devices, mobile computing devices, mobile communicationdevices (e.g., smart phone), and/or other computing devices. Forexample, computing system 600 may include and/or be included incomputing device 104 and/or head-mounted display device 106 of FIGS. 1,3, and 5.

Computing system 600 includes a logic machine 602 and a storage machine604. Computing system 600 may optionally include a display subsystem606, input subsystem 608, communication subsystem 610, and/or othercomponents not shown in FIG. 6.

Logic machine 602 includes one or more physical devices configured toexecute instructions. For example, the logic machine may be configuredto execute instructions that are part of one or more applications,services, programs, routines, libraries, objects, components, datastructures, or other logical constructs. Such instructions may beimplemented to perform a task, implement a data type, transform thestate of one or more components, achieve a technical effect, orotherwise arrive at a desired result.

The logic machine may include one or more processors configured toexecute software instructions. Additionally or alternatively, the logicmachine may include one or more hardware or firmware logic machinesconfigured to execute hardware or firmware instructions. Processors ofthe logic machine may be single-core or multi-core, and the instructionsexecuted thereon may be configured for sequential, parallel, and/ordistributed processing. Individual components of the logic machineoptionally may be distributed among two or more separate devices, whichmay be remotely located and/or configured for coordinated processing.Aspects of the logic machine may be virtualized and executed by remotelyaccessible, networked computing devices configured in a cloud-computingconfiguration.

Storage machine 604 includes one or more physical devices configured tohold instructions executable by the logic machine to implement themethods and processes described herein. When such methods and processesare implemented, the state of storage machine 604 may betransformed—e.g., to hold different data.

Storage machine 604 may include removable and/or built-in devices.Storage machine 604 may include optical memory (e.g., CD, DVD, HD-DVD,Blu-Ray Disc, etc.), semiconductor memory (e.g., RAM, EPROM, EEPROM,etc.), and/or magnetic memory (e.g., hard-disk drive, floppy-disk drive,tape drive, MRAM, etc.), among others. Storage machine 604 may includevolatile, nonvolatile, dynamic, static, read/write, read-only,random-access, sequential-access, location-addressable,file-addressable, and/or content-addressable devices.

It will be appreciated that storage machine 604 includes one or morephysical devices. However, aspects of the instructions described hereinalternatively may be propagated by a communication medium (e.g., anelectromagnetic signal, an optical signal, etc.) that is not held by aphysical device for a finite duration.

Aspects of logic machine 602 and storage machine 604 may be integratedtogether into one or more hardware-logic components. Such hardware-logiccomponents may include field-programmable gate arrays (FPGAs), program-and application-specific integrated circuits (PASIC/ASICs), program- andapplication-specific standard products (PSSP/ASSPs), system-on-a-chip(SOC), and complex programmable logic devices (CPLDs), for example.

The terms “module,” “program,” and “engine” may be used to describe anaspect of computing system 600 implemented to perform a particularfunction. In some cases, a module, program, or engine may beinstantiated via logic machine 602 executing instructions held bystorage machine 604. It will be understood that different modules,programs, and/or engines may be instantiated from the same application,service, code block, object, library, routine, API, function, etc.Likewise, the same module, program, and/or engine may be instantiated bydifferent applications, services, code blocks, objects, routines, APIs,functions, etc. The terms “module,” “program,” and “engine” mayencompass individual or groups of executable files, data files,libraries, drivers, scripts, database records, etc.

It will be appreciated that a “service”, as used herein, is anapplication program executable across multiple user sessions. A servicemay be available to one or more system components, programs, and/orother services. In some implementations, a service may run on one ormore server-computing devices.

When included, display subsystem 606 may be used to present a visualrepresentation of data held by storage machine 604. This visualrepresentation may take the form of a graphical user interface (GUI). Asthe herein described methods and processes change the data held by thestorage machine, and thus transform the state of the storage machine,the state of display subsystem 606 may likewise be transformed tovisually represent changes in the underlying data. For example,responsive to instructions from a logic machine of head-mounted displaydevice 106 of FIG. 1, the state of a see-through display may betransformed to display an application launcher thereon responsive torecognition of a user trigger. Display subsystem 606 may include one ormore display devices utilizing virtually any type of technology. Suchdisplay devices may be combined with logic machine 602 and/or storagemachine 604 in a shared enclosure, or such display devices may beperipheral display devices.

When included, input subsystem 608 may comprise or interface with one ormore user-input devices such as a keyboard, mouse, touch screen, or gamecontroller. In some embodiments, the input subsystem may comprise orinterface with selected natural user input (NUI) componentry. Suchcomponentry may be integrated or peripheral, and the transduction and/orprocessing of input actions may be handled on- or off-board. Example NUIcomponentry may include a microphone for speech and/or voicerecognition; an infrared, color, stereoscopic, and/or depth camera formachine vision and/or gesture recognition; a head tracker, eye tracker,accelerometer, and/or gyroscope for motion detection and/or intentrecognition; as well as electric-field sensing componentry for assessingbrain activity.

When included, communication subsystem 610 may be configured tocommunicatively couple computing system 600 with one or more othercomputing devices. Communication subsystem 610 may include wired and/orwireless communication devices compatible with one or more differentcommunication protocols. As non-limiting examples, the communicationsubsystem may be configured for communication via a wireless telephonenetwork, or a wired or wireless local- or wide-area network. In someembodiments, the communication subsystem may allow computing system 600to send and/or receive messages to and/or from other devices via anetwork such as the Internet.

FIG. 7 shows a non-limiting example of the HMD device 106 of FIGS. 1 and3 in the form of a pair of wearable glasses with a see-through display702. It will be appreciated an HMD device may take any other suitableform in which a transparent, semi-transparent, and/or non-transparentdisplay is supported in front of a viewer's eye or eyes. Further,embodiments described herein may be used with any other suitablecomputing device, including but not limited to mobile computing devices,laptop computers, desktop computers, tablet computers, other wearablecomputers, etc.

The HMD device 106 includes a see-through display 702 and a controller704. The see-through display 702 may enable images such as holographicobjects to be delivered to the eyes of a wearer of the HMD device. Thesee-through display 702 may be configured to visually augment anappearance of a real-world, physical environment to a wearer viewing thephysical environment through the transparent display. In one example,the display may be configured to display one or more UI objects on agraphical user interface. In some embodiments, the UI objects presentedon the graphical user interface may be virtual objects overlaid in frontof the real-world environment. Likewise, in some embodiments, the UIobjects presented on the graphical user interface may incorporateelements of real-world objects of the real-world environment seenthrough the see-through display 702. In other examples, the display maybe configured to display one or more other graphical objects, such asvirtual objects associated with games, videos, or other visual content.

Any suitable mechanism may be used to display images via the see-throughdisplay 702. For example, the see-through display 702 may includeimage-producing elements located within lenses 706 (such as, forexample, a see-through Organic Light-Emitting Diode (OLED) display). Asanother example, the see-through display 702 may include a displaydevice (such as, for example a liquid crystal on silicon (LCOS) deviceor OLED microdisplay) located within a frame of HMD device 106. In thisexample, the lenses 706 may serve as, or otherwise include, a lightguide for delivering light from the display device to the eyes of awearer. Such a light guide may enable a wearer to perceive a 3Dholographic image located within the physical environment that thewearer is viewing, while also allowing the wearer to view physicalobjects in the physical environment, thus creating a mixed realityenvironment.

The HMD device 106 may also include various sensors and related systemsto provide information to the controller 704. Such sensors may include,but are not limited to, one or more inward facing image sensors 708 aand 708 b, one or more outward facing image sensors 710, an inertialmeasurement unit (IMU) 712, and one or more microphones 720. The one ormore inward facing image sensors 708 a, 708 b may be configured toacquire image data in the form of gaze tracking data from a wearer'seyes (e.g., sensor 708 a may acquire image data for one of the wearer'seye and sensor 708 b may acquire image data for the other of thewearer's eye). The HMD device may be configured to determine gazedirections of each of a wearer's eyes in any suitable manner based onthe information received from the image sensors 708 a, 708 b. Forexample, one or more light sources 714 a, 714 b, such as infrared lightsources, may be configured to cause a glint of light to reflect from thecornea of each eye of a wearer. The one or more image sensors 708 a, 708b may then be configured to capture an image of the wearer's eyes.Images of the glints and of the pupils as determined from image datagathered from the image sensors 708 a, 708 b may be used by thecontroller 704 to determine an optical axis of each eye. Using thisinformation, the controller 704 may be configured to determine adirection the wearer is gazing. The controller 704 may be configured toadditionally determine an identity of a physical and/or virtual objectat which the wearer is gazing.

The one or more outward facing image sensors 710 may be configured toreceive physical environment data from the physical environment in whichthe HMD device 106 is located. Data from the outward facing imagesensors 710 may be used to detect movements within a field of view ofthe display 702, such as gesture-based inputs or other movementsperformed by a wearer or by a person or physical object within the fieldof view. In one example, data from the outward facing image sensors 710may be used to detect a selection input performed by the wearer of theHMD device, such as a gesture (e.g., a pinching of fingers, closing of afist, etc.), that indicates selection of a UI object displayed on thedisplay device. Data from the outward facing sensors may also be used todetermine direction/location and orientation data (e.g. from imagingenvironmental features) that enables position/motion tracking of the HMDdevice 106 in the real-world environment.

The IMU 712 may be configured to provide position and/or orientationdata of the HMD device 106 to the controller 704. In one embodiment, theIMU 712 may be configured as a three-axis or three-degree of freedomposition sensor system. This example position sensor system may, forexample, include three gyroscopes to indicate or measure a change inorientation of the HMD device 106 within 3D space about three orthogonalaxes (e.g., x, y, z) (e.g., roll, pitch, yaw). The orientation derivedfrom the sensor signals of the IMU may be used to display, via thesee-through display, one or more virtual UI objects in three degrees offreedom.

In another example, the IMU 712 may be configured as a six-axis orsix-degree of freedom position sensor system. Such a configuration mayinclude three accelerometers and three gyroscopes to indicate or measurea change in location of the HMD device 106 along the three orthogonalaxes and a change in device orientation about the three orthogonal axes.In some embodiments, position and orientation data from the outwardfacing image sensors 710 and the IMU 712 may be used in conjunction todetermine a position and orientation of the HMD device 106.

The HMD device 106 may also support other suitable positioningtechniques, such as GPS or other global navigation systems. Further,while specific examples of position sensor systems have been described,it will be appreciated that any other suitable position sensor systemsmay be used. For example, head pose and/or movement data may bedetermined based on sensor information from any combination of sensorsmounted on the wearer and/or external to the wearer including, but notlimited to, any number of gyroscopes, accelerometers, inertialmeasurement units, GPS devices, barometers, magnetometers, cameras(e.g., visible light cameras, infrared light cameras, time-of-flightdepth cameras, structured light depth cameras, etc.), communicationdevices (e.g., WIFI antennas/interfaces), etc.

Continuing with FIG. 7, the controller 704 may be configured to recordmultiple eye gaze samples over time based on information detected by theone or more inward facing image sensors 708 a, 708 b. For each eye gazesample, eye tracking information and, in some embodiments, head trackinginformation (from image sensors 710 and/or IMU 712) may be used toestimate an origin point and a direction vector of that eye gaze sampleto produce an estimated location at which the eye gaze intersects thesee-through display. Examples of eye tracking information and headtracking information used to determine an eye gaze sample may include aneye gaze direction, head orientation, eye gaze velocity, eye gazeacceleration, change in angle of eye gaze direction, and/or any othersuitable tracking information. In some embodiments, eye gaze trackingmay be recorded independently for both eyes of the wearer of the HMDdevice 106.

As mentioned above, the HMD device 106 may also include one or moremicrophones, such as microphone 720, that capture audio data. In someexamples, the one or more microphones 720 may comprise a microphonearray including two or more microphones. For example, the microphonearray may include four microphones, two positioned above the right lensand two positioned above the left lens of the HMD device. Further, audiooutputs may be presented to the wearer via one or more speakers, such asspeaker 722. In some embodiments, the microphone system may beconfigured to provide audio feedback indicating selection of a UI objectpresented on a graphical user interface displayed on the see-throughdisplay.

The controller 704 may include a logic machine and a storage machine,discussed in more detail above with respect to FIG. 6, in communicationwith the various sensors and display of the HMD device. In one example,the storage machine may include instructions that are executable by thelogic machine to provide sound for a displayed virtual object to one ormore external speakers, as described herein.

Another example provides a method comprising recognizing, via aneye-tracking camera, a gaze target, responsive to a user trigger,displaying, via a display, an application launcher at the gaze target,the application launcher presenting a plurality of applicationsselectable for launching, recognizing a user selection of one of theplurality of applications, and replacing, via the display, theapplication launcher with the selected one of the plurality ofapplications at the gaze target. In such an example, replacing theapplication launcher with the selected one of the plurality ofapplications may additionally or alternatively comprise displaying agraphical user interface element of the selected one of the plurality ofapplications at the gaze target. In such an example, the user triggermay additionally or alternatively comprise a gesture recognized via oneor more of the eye-tracking camera and another imaging device. In suchan example, the user trigger may additionally or alternatively comprisea voice command recognized via a microphone. In such an example, theuser trigger may additionally or alternatively comprise a sustainedgaze, and the method may additionally or alternatively further compriserecognizing the user trigger responsive to recognizing, via theeye-tracking camera, that a user gaze is directed to the gaze target fora duration that is greater than a trigger threshold. In such an example,the method may additionally or alternatively further comprise, prior todisplaying the application launcher, displaying a positional preview ofthe application launcher at the gaze target. In such an example, themethod may additionally or alternatively further comprise replacing, viathe display, the positional preview of the application launcher with theapplication launcher responsive to a confirmation input. In such anexample, the display may additionally or alternatively comprise asee-through display of a head-mounted display device and theeye-tracking camera may additionally or alternatively be integrated intothe head-mounted display device. In such an example, the display mayadditionally or alternatively comprise a see-through display of ahead-mounted display device and the user trigger may additionally oralternatively be recognized based upon analysis of an image captured byan imaging device remote from the head-mounted display device. Any orall of the above-described examples may be combined in any suitablemanner in various implementations.

Another example provides a head-mounted display device comprising aneye-tracking camera, a see-through display, a logic machine, and astorage machine holding instructions executable by the logic machine torecognize, via the eye-tracking camera, a gaze target, responsive to auser trigger, display, via the see-through display, an applicationlauncher at the gaze target, the application launcher presenting aplurality of applications selectable for launching, recognize a userselection of one of the plurality of applications, and replace, via thedisplay, the application launcher with the selected one of the pluralityof applications at the gaze target. In such an example, the head-mounteddisplay device may additionally or alternatively further comprise aninput subsystem, and the user trigger may additionally or alternativelybe detected via the input subsystem. In such an example, the usertrigger may additionally or alternatively comprise a gesture recognizedvia one or more of the head-mounted display device and a camera remotefrom the head-mounted display device. In such an example, thehead-mounted display device may additionally or alternatively furthercomprise a microphone, and the user trigger may additionally oralternatively comprise a voice command recognized via the microphone. Insuch an example, the user trigger may additionally or alternativelycomprise a sustained gaze, and the instructions may additionally oralternatively be further executable to recognize the sustained gazeresponsive to determining, via the eye-tracking camera, that the gazetarget is maintained for a duration that is greater than a triggerthreshold. In such an example, replacing the application launcher withthe selected one of the plurality of applications may additionally oralternatively comprise displaying, via the see-through display, agraphical user interface element of the selected one of the plurality ofapplications at the gaze target. In such an example, the instructionsmay additionally or alternatively be further executable to, prior todisplaying the application launcher, display, via the see-throughdisplay, a positional preview of the application launcher at the gazetarget. In such an example, the instructions may additionally oralternatively be further executable to replace, via the see-throughdisplay, the positional preview of the application launcher with theapplication launcher responsive to a confirmation input. Any or all ofthe above-described examples may be combined in any suitable manner invarious implementations.

Another example provides a head-mounted display device comprising aneye-tracking camera, an input subsystem, a see-through display, a logicmachine, and a storage machine holding instructions executable by thelogic machine to recognize, via the eye-tracking camera, a gaze target,responsive to a user trigger detected via the input subsystem, display,via the see-through display, a positional preview of an applicationlauncher at the gaze target, responsive to a confirmation input detectedvia the input subsystem, replacing, via the see-through display, thepositional preview of the application launcher with the applicationlauncher presenting a plurality of applications selectable forlaunching, recognize a user selection of one of the plurality ofapplications, and replace, via the display, the application launcherwith the selected one of the plurality of applications at the gazetarget. In such an example, the user trigger may additionally oralternatively comprise a sustained gaze, and the instructions mayadditionally or alternatively be further executable to recognize thesustained gaze responsive to determining, via the eye-tracking camera,that the gaze target is maintained for a duration that is greater than atrigger threshold. In such an example, the user trigger may additionallyor alternatively comprise one or more of a gesture and a voice commandrecognized via the input subsystem. Any or all of the above-describedexamples may be combined in any suitable manner in variousimplementations.

It will be understood that the configurations and/or approachesdescribed herein are exemplary in nature, and that these specificembodiments or examples are not to be considered in a limiting sense,because numerous variations are possible. The specific routines ormethods described herein may represent one or more of any number ofprocessing strategies. As such, various acts illustrated and/ordescribed may be performed in the sequence illustrated and/or described,in other sequences, in parallel, or omitted. Likewise, the order of theabove-described processes may be changed.

The subject matter of the present disclosure includes all novel andnon-obvious combinations and sub-combinations of the various processes,systems and configurations, and other features, functions, acts, and/orproperties disclosed herein, as well as any and all equivalents thereof.

The invention claimed is:
 1. A method, comprising: displaying, via asee-through display of a head-mounted display device, one or moreaugmentation elements; recognizing, via an eye-tracking camera of thehead-mounted display device, a gaze target on a surface in a real-worldenvironment viewed through the see-through display; responsive to a usertrigger, displaying, via the see-through display, an applicationlauncher overlaid on the surface of the real-world environment at thegaze target, the application launcher having a world-locked positionrelative to the real-world environment such that the applicationlauncher does not appear to move relative to the real-world environmentwhen the head-mounted display device moves, the application launcherpresenting a plurality of applications selectable for launching within agraphical boundary of the application launcher that occupies a region ofthe see-through display to allow for the one or more augmentationelements and one or more real-world objects in the real-worldenvironment to be viewed outside of the graphical boundary of theapplication launcher through the see-through display; recognizing a userselection of one of the plurality of applications; and replacing, viathe see-through display, the application launcher with the selected oneof the plurality of applications by displaying a graphical userinterface of the selected one of the plurality of applications at theworld-locked position overlaid on the surface of the real-worldenvironment where the application launcher was previously displayed suchthat the graphical user interface of the selected one of the pluralityof applications is presented within the graphical boundary of theapplication launcher.
 2. The method of claim 1, wherein the user triggercomprises a gesture recognized via one or more of the eye-trackingcamera and another imaging device of the head-mounted display device. 3.The method of claim 1, wherein the user trigger comprises a voicecommand recognized via a microphone of the head-mounted display device.4. The method of claim 1, wherein the user trigger comprises a sustainedgaze, the method further comprising recognizing the user triggerresponsive to recognizing, via the eye-tracking camera, that a user gazeis directed to the gaze target for a duration that is greater than atrigger threshold.
 5. The method of claim 1, further comprising, priorto displaying the application launcher, displaying, via the see-throughdisplay, a positional preview of the graphical boundary of theapplication launcher overlaid on the surface of the real-worldenvironment at the gaze target, the positional preview providing anindication of position, size, and/or shape of the graphical boundarywithin which the application launcher will be displayed.
 6. The methodof claim 5, further comprising replacing, via the see-through display,the positional preview of the graphical boundary of the applicationlauncher with the application launcher presenting the plurality ofapplications selectable for launching within the graphical boundary ofthe application launcher responsive to a confirmation input.
 7. Themethod of claim 1, wherein the application launcher is displayed, viathe see-through display, with a center point of the application launcherpositioned at the gaze target on the surface of the real-worldenvironment, and wherein the graphical user interface of the selectedone of the plurality of applications is displayed, via the see-throughdisplay, with a center point of the graphical user interface positionedat a same location where the center point of the application launcherwas previously displayed.
 8. A head-mounted display device comprising:an eye-tracking camera; a see-through display; a logic machine; and astorage machine holding instructions executable by the logic machine to:display, via the see-through display, one or more augmentation elements;recognize, via the eye-tracking camera, a gaze target on a surface in areal-world environment viewed through the see-through display;responsive to a user trigger, display, via the see-through display, anapplication launcher overlaid on the surface of the real-worldenvironment at the gaze target, the application launcher having aworld-locked position relative to the real-world environment such thatthe application launcher does not appear to move relative to thereal-world environment when the head-mounted display device moves, theapplication launcher presenting a plurality of applications selectablefor launching within a graphical boundary of the application launcherthat occupies a region of the see-through display to allow for the oneor more augmentation elements and one or more real-world objects in thereal-world environment to be viewed outside of the graphical boundary ofthe application launcher through the see-through display; recognize auser selection of one of the plurality of applications; and replace, viathe see-through display, the application launcher with the selected oneof the plurality of applications by displaying, in place of theapplication launcher, a graphical user interface of the selected one ofthe plurality of applications at the world-locked position overlaid onthe surface of the real-world environment where the application launcherwas previously displayed such that the graphical user interface of theselected one of the plurality of applications is presented within thegraphical boundary of the application launcher.
 9. The head-mounteddisplay device of claim 8, further comprising an input subsystem,wherein the user trigger is detected via the input subsystem.
 10. Thehead-mounted display device of claim 8, wherein the user triggercomprises a gesture recognized via one or more of the head-mounteddisplay device and a camera remote from the head-mounted display device.11. The head-mounted display device of claim 8, further comprising amicrophone, and wherein the user trigger comprises a voice commandrecognized via the microphone.
 12. The head-mounted display device ofclaim 8, wherein the user trigger comprises a sustained gaze, andwherein the instructions are further executable to recognize thesustained gaze responsive to determining, via the eye-tracking camera,that the gaze target is maintained for a duration that is greater than atrigger threshold.
 13. The head-mounted display device of claim 8, theinstructions further executable to, prior to displaying the applicationlauncher, display, via the see-through display, a positional preview ofthe graphical boundary of the application launcher overlaid on thesurface of the real-world environment at the gaze target, the positionalpreview providing an indication of position, size, and/or shape of thegraphical boundary within which the application launcher will bedisplayed.
 14. The head-mounted display device of claim 13, theinstructions further executable to replace, via the see-through display,the positional preview of the graphical boundary of the applicationlauncher with the application launcher presenting the plurality ofapplications selectable for launching within the graphical boundary ofthe application launcher responsive to a confirmation input.
 15. Ahead-mounted display device comprising: an eye-tracking camera; an inputsubsystem; a see-through display; a logic machine; and a storage machineholding instructions executable by the logic machine to: display, viathe see-through display, one or more augmentation elements; recognize,via the eye-tracking camera, a gaze target on a surface in a real-worldenvironment viewed through the see-through display; responsive to a usertrigger detected via the input subsystem, display, via the see-throughdisplay, a positional preview of a graphical boundary of an applicationlauncher overlaid on the surface of the real-world environment at thegaze target, the positional preview of the graphical boundary of theapplication launcher having a world-locked position relative to thereal-world environment such that the positional preview of the graphicalboundary of the application launcher does not appear to move relative tothe real-world environment when the head-mounted display device moves,the positional preview of the graphical boundary of the applicationlauncher providing an indication of position, size, and/or shape of thegraphical boundary within which the application launcher will bedisplayed, and the positional preview of the graphical boundary of theapplication launcher occupying a region of the see-through display toallow for the one or more augmentation elements and one or morereal-world objects in the real-world environment to be viewed outside ofthe graphical boundary of the application launcher through thesee-through display, responsive to a confirmation input detected via theinput subsystem, replacing, via the see-through display, the positionalpreview of the graphical boundary of the application launcher with theapplication launcher by displaying, in place of the positional previewof the graphical boundary of the application launcher, the applicationlauncher overlaid on the surface of the real-world environment at thegaze target where the positional preview of the graphical boundary ofthe application launcher was previously displayed, the applicationlauncher having a world-locked position relative to the real-worldenvironment such that the application launcher does not appear to moverelative to the real-world environment when the head-mounted displaydevice moves, the application launcher presenting a plurality ofapplications selectable for launching within the graphical boundary ofthe application launcher that occupies the region of the see-throughdisplay to allow for the one or more augmentation elements and the oneor more real-world objects in the real-world environment to be viewedoutside of the graphical boundary of the application launcher throughthe see-through display; recognize a user selection of one of theplurality of applications; and replace, via the see-through display, theapplication launcher with the selected one of the plurality ofapplications by displaying, in place of the application launcher, agraphical user interface of the selected one of the plurality ofapplications overlaid on the surface of the real-world environment wherethe application launcher was previously displayed such that thegraphical user interface of the selected one of the plurality ofapplications is presented within the graphical boundary of theapplication launcher.
 16. The head-mounted display device of claim 15,wherein the user trigger comprises a sustained gaze, and wherein theinstructions are further executable to recognize the sustained gazeresponsive to determining, via the eye-tracking camera, that the gazetarget is maintained for a duration that is greater than a triggerthreshold.
 17. The head-mounted display device of claim 15, wherein theuser trigger comprises one or more of a gesture and a voice commandrecognized via the input subsystem.